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ABSTRACT 


The design, development and _ use of cost-effective 
computer networks require information about system. behavior 
given a variety of network structures and operational 
policies. Because computer networks..are. complex systems whose 
behavior is generally not intuitively understood, there is a 
.need for system analysis tools.to provide,.a wide range of 
performance information. 

This. thesis describes. a simulation system that generates 
behavioral information for a class of minicomputer network 
systems. This simulation system is modularly designed with 
modules for network modelling, specification of the network 
processing load, and simulation. (a. disorete:. event. simulator) . 
The network modelling done with the simulation system is based 
on a_ general purpose discrete. medelling; discipline. Fiexible 
network model building blocks made from the basic modelling 
discipline structures are provided ta.the, simulation, system 
user. To prepare a simulation experiment the user assembles a 
network model from the. building blocks and specifies a network 
processing load. To generate performance information the 
network model and load specification: are. input to the 
simulator along with simulation control parameters. On 
completion of the simulation experiment the generated 
performance information is output in a palatable form to the 
user. Overall this simulation system,.:is...a-convenient. and 
flexible system analysis tool for minicomputer networks. 
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CHAPTER 1 


A network is an interconnected or daverteinced group or 
system. Network systems have been developed for railroads, 
telephone communications, mail services and electric power. In 
each case the development of a network from previously 
independent components has brought improved service at lower 
cost. 

Based on this tradition of success in the development of 
network systems, it is not surprising to see a worldwide move 
towards the use of data processing-oriented communications 
networks (ie. computer networks). These networks are generally 
formed by the interconnection, via communications links, of 
computer systems. Among the goals in the move to -develop 
computer networks are to provide information  sécurity, 
processing reliability and cost-effective proeéssing services. 

The physical disribution of protessing services in a 
computer network can be exploited in providing the appropr iate 
degree of information security to interacting processing tasks 
with different security requirements. One way to do this’is to 
customize each node in a network to provide a different level 
of security (in terms of operating system safeguards, physical 
protection of the computer, etc.). Then processing tasks can 
be assigned to the computer system with the appropriate 
security and at the same time retain a communications 
capability to tasks with other security requirements. The 


network approach to security can be more effective than 


Placing all tasks within the security structure of a single 
computer. 

The network approach to reliability is based on the 
redundant processing capability inherent in a computer 
network. If one computer system in a network fails, the other 
systems in the network can be used to dynamically recover from 
the failure. Optitally, the processing load of the system that 
failed can be shifted without penalty to other systems and the 
globally observed operation of the network is unchanged. In 
most cases though, it must be expected that a degradation of 
network performance will occur after a single system failure. 
This gradual degradation of services, made possible through 
networking, is one way to avoid abrupt and total 
discontinuations of service (ie. when the processor fails in a 
computer systéin with no backup processor). 

Perhaps the most important feature of computer networks 
is the potential to provide cost-efféctive processing 
Services. There are a number of ways to realize this 
potential. Staring of hardware, software and information 
resources can. be done with a network of computers of different 
characteristics. A user can access costly resources without 
having those resources associated with the locally available 
computer. system (eg. an expensive peripheral need be located 
on only one system in a network). Another way a network can be 
cost-effective is by allowing local data processing at the 


site of data acquisition and use, while at the same time 


Maintaining data links to remote processing centers (eg. a 
bank “with many branches would be a potential network 
application). This arrangement yields significant savings in 
communications costs over a completely centralized processing 
facility. Yet another cost-ef fective Gharacteristic of 
networks is the allowance for a “gradual and consistent 
expansion of the processing capabilities as they are required 
by an application. Networking can offer a range of processing 
capabilities more expansive than is offered by any single 
computer line. 

While computer networks have great potential in providing 
a variety of data processing capabilities and services, they 
also pose difficult design and implementation problems. A 
network system is inherently more complex than the ‘individual 
components that are linked together in the system. Ih so far 
as the behavior of a computer system is difficult to predict 
for a given user environment, prediction of the behavior of a 
computer network system can be mind boggling. But this 
behavior must be anticipated to some extent in order to 
develop effective data processing systems. 

There are two primary phases in the development of 
computer networks. First, commun ications hardware and 
software must be developed to allow a computer system to 
function in a network environment. Modems, communications 
processors, programs to maintain communications protocols, and 


a host of other computer communications components must be 
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designed and implemented. The second phase of network 
development involves selecting the appropriate computer and 
communications options in order to fill specific data 
processing needs. ° During both phases a network designer needs 
insight into the behavior of network systems. | 7 

System analysis tools provide information to supplement a 
designer’s intuitive understanding of network behavior. These 
tools include hardware and software monitors for sgyeten 
Measurement, analytic models, and system simulators. Most 
analysis tools are restricted in their application to 
particular aspects of network behavior. The complexity of 
network systems, coupled with the inherent Lipitations of the 
analysis tools cause this restriction. Therefore, in order to 
cope with new types of networks, there is a continuing need 
for new network analysis tools. i | 

This thesis describes the development of a simulation 
system to be used for analysis of the behavior: Of a 
minicomputer network system. The following chapters trace the 
development of the Minicomputer Network Simulation system 
(MNSS). Chapters 2,3 and 4 evase the research into, and the 
design of a flexible network model of a particular class of 
minicomputer system (designated Hewlett-Packard Computer 
systems or HPC systems). Chapters 5 and 6 descr ibe the design 
and implementation of an effective interactive eimulator that 
uses HPC network models. Chapter 7 provides a summary of some 


simulation experiments designed to demonstrate the 


ll 


Capabilities of MNSS. Chapter 8 reviews the progress made by 


this thesis project and the potential for further work. 
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CHAPTER 2 
BACKGROUND 


There is a rich background of work concerned with 
computer system analysis. From this background are drawn the 
techniques that are used to develop computer network analysis 
tools. These tools include analytical models, performance 
monitors, and simulation systems. Analytical models are 
usually very efficient but are limited in application to 
relatively simple, well characterized systems. The use of 
performance monitors in network analysis is aimed at getting 
very accurate and specific information. A fundamental 
limitation of performance monitors is the necessity to have 
available a working version of the system to be analyzed. 
Simulation systems can be used to do system analysis at any 
stage in system development. In addition, a network system can 
be analyzed in as much detail as is necessary. The efficiency 
of a simulation system in analyzing network behavior is in 
general lower than that of an analytical model and the 
behavioral information derived with simulation is less 
accurate than that gathered by a performance monitor. 
Simulation systems, performance monitors, and analytic models, 
by having different operational characteristics, provide the 
capabilities for a range of computer network analysis at 
various stagés in system development and use. 


The following sections review the most pertinent 
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background to the development of MNSS. This includes 
simulation modelling techniques and the implementation ,and,.use 
of computer network simulators. The principles behind the 
design of simulation systems in general, and MNSS in 
. particular, can be distilled from the sea of..work that has 
been reported. This work can be characterized into two sroupe 
corresponding to the principle stages in.the MNSS development 
process. Initially models of minicomputer network systems had 
to be designed to describe all significant features. Then an 
effective simulator had to be designed..and implemented. that 


uses these network models. 
2.1 Simulation Modelling 


A representative model of a system is required to do 
simulation. Models used for simulation can be divided into two 
general classes: structural models and functional models [27]. 
Individual system components. and their connections are 
. represented in a structural model. This.level of detail is 
appropriate for tasks such as logic simulation of digital 
systems [5]. A functional model provides a mare abstract 
representaion of a system than a structural model. It 
describes how a system operates and can be used for 
mathematical or empirical system analysis. Extensive use of 
functional models, in the form of flowchart. models [2], 


finite-state models [10], parallel nets [22] ana. queuing 
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models [23], is made in simulating complex hardware and 
 goftware computer system structures, 

The effectiveness of a simulation model can be gauged by 
its ability to represent all significant system features and 
“at the ‘same time minimize the effort required to use it for 
system simulation. In the process of constructing a 
simulation model there are a number of factors that impact its 
effectiveness. These factors involve characteristics of the 
modelling language used to describe the model, and various 


qualities of the model. Specifically they include: 


1. the flexibility of the modelling language (ie. the model 
description language) in abstracting real system 
structures, 

2. the efficiency of the modelling language in a simulation 
context, 

3. the ease with which a model can be adapted to changes in 
the structure of the system modelled, 

4. the level of detail that is used in a model to represent 
significant system features (that may be at different 
levels of detail in different parts of the system 
structure), and 


5. the effort required to verify and validate the model. 


Work is continually being done on modelling languages and 


méthods in order to deal with these factors. — 
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URS + OR eae? 


A number of approaches have evolved for developing 
effective modelling languages. One_ approach is to nave the 
modelling language be identical to the simulator programming 
language (ie. the programming language used to iGesiewene the 
mechanism for simulating the mode 1) (11). While this approach 
can yield efficient simulation processing, the process of 
abstracting a system in this type of ‘modelling language may be 
very difficult. An ever expanding set of instructions may be 
needed to handle new and evolving system structures. A second 
approach to developing a mode11 ing language is ‘to have only a 
partial correspondence with the simulator programming language 
(eg. using a simulation language sGah ae SIMULA or GASP to 
implement a modelling language) {16]. With ehié approach the 
simulator programming language performs simulation 
housekeeping chores and well defined activities such — table 
construction. System representaion in the modelling language 
can be done with more ease than in the first approach, but the 
conversion to the simulator language is ware complex. Another 
approach is to use a general puepeee: Tanguase (FORTRAN, APL) 
[23] to implement the simulator, and have a separate modelling 
language which specifically deals with the structures of the 
system to be modelled. Representing system structures in the 
modelling aanguase becomes increasingly straightforward | at the 
cost of more effort to implement the simulator. 

Whatever the relationship between the mode 11 ing language 


and the simulator language, it is advantageous to have the 
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modelling language provide convenient model description 
building biocks [{7]. The building blocks may vary from the 56 
instructions “in GPSS-5 to the two principle structures in 
DYNAMO. By having an effective set of building blocks a 
modelling language can be used to represent a range of 
different types of systems; a modeller can be spared the task 
of mastering several limited (special purpose) languages. In 
addition a@ model constructed using a well known set of 
building blocks can be understood by more people than one 
constructed using specialized, one-of-a-kind structures. 

The complexity of a modelling Language can be gauged by 
the number and types of the associated bulldine blocks. In 
general a modelling language becomes acts difficult to use and 
understand with larger numbers of building blocks. The degree 
of correspondence “between building blocks and real avaten 
structures also influences the complexity of language use. A 
modelling language for a limited and well defined set of 
systems can incorporate just a few very specific building 
blocks and be very easy to use. But a modelling language used 
for general classes of systems must either have a great numbe r 
of specific building blocks or a smaller number that are very 
general. In either case the generalized modelling language is 
harder to use than a specific modelling language. 

Given a particular modelling language, the effectiveness 
of a model is affected by the technique of the modeller. 


Modelling techniques that promote the development of effective 
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models include top-down development, modular design, and 
submodel organization. 

Top-down development is extensively used in software 
development to facilitate debugging, functional verification, 
and maintenance. It is not surprising that top-down 
development has been constructively applied to simulation 
modelling [6]. Design and verification activities can be 
integrated together during top-down implementation of a 
simuation model. Since the conceptual model is usually derived 
in a top-down manner, this is a natural and efficient 
approach. 

Modular design allows a modeller to adapt a model to 
changes in a system structure by making localized changes to 
the model. The model can be effectively used in simulations in 
which both parametric and structural system characteristics 
vary [24]. Parametric characteristics are subject to variation 
by changing single, usually numeric values; structural 
characteristics involve functional aspects of a system and are 
varied by changing the functional modules that make up the 
model description. For example, with a modularly designed 
network simulation model, not only can the line speeds and 
message frequencies be varied (parametric characteristics), 
but also the network configuration, communications protocols 
and processor scheduling algorithms (structural 
characteristics) can be changed. Modular design makes a model 


easier to develop and adaptable to structural variation for 
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simulation studies. 

Submodel organization is a hign level application of 
modular design (several modules may be included in a submode1) 
which is facilitated by the decomposition of a system into 
several system components or subsystems. The simulation of a 
system is restricted by time and cost limitations when a model 
is developed at avery fine level (low level) of detail. By 
using a submodel organization, dif ferent levels of detail can 
be used in individual submodels to ease simulation 
restrictions while maintaining the accuracy of the model 
representation [21]. Submodel representaions may range from a 
decision table to a highly detailed model (subject to 
simulation apart from the rest of the model). By using a 
decision table or a simple function to directly generate 
submodel outputs from the inpute: dhen enslgeical results, 
measurement data, and previous simulation results can be 
utilized to reduce needless simulation. Where a detailed 
representaion of a subsystem is needed for modelling accuracy, 
simulation on the desired low level of detail can be limited 
to che specific submodel. Excessive simulation is avoided by 
not having to model all subsystems at the sane low level of 
detail. | | 

A submodel organization can also be exploited in model 
validation [3]. Each submodel can be exercised and compared to 
the corresponding real subsystem. The submodels will be less 


complex than the complete model and easier to validate. Once 
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each submodel is shown to operate: correctly then submodel 
_interactions can be proven correct (or redefined to become 
correct). This stepwise approach to validation avoids many of 
the problems of working with large and. very complex models. It 
is especially valuable where there is a model comprised of 
Many duplicate submodels (eg. a homogeneous network model with 


each node being a submodel). 
2.2 Simulation System Design 


A computer simulation system provides facilities to build 
a simulation model, takes as input a workload ese ripe sony and 
uses a simulator mechanism to perform experiments. The 
simulation model and workload "description determine the 
behavioral information that is generated and recorded for 
‘performance analysis by the simulator. Once a simulation model 
or the building blocks necessary to form a set of related 
simulation models have been developed, the next steps in the 
development of a simulation system are the design and 
implementaion of an efficient simulator. and a comfortable 
user-simulator interface. | 

Computer network simulators can be classified according 
to the form of the workload description they ieapamong: the 
most common classes are stochastic and trace driven (27). For 
a stochastic simulator ihe wotkiess is descr ibed by 
probabilistic distributions; resource demands are generated as 
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random variables from these distributions. In contrast, a 
trace driven simulator operates with a workload represented as 
a deterministic sequence of _ resource demands. Traée driven 
Bimulation is very useful in tuning a system for well defined 
applications. It cannot be used for an application that cannot 
be represented accurately by a deterministic sequence of 
resource demands or where exact resource demand information is 
not available; in these cases stochastic simulation may be 
appropriate. Both types 6E-eieutation can be used to examine 
the performance of new system designs, alternative system 
configurations and vesckice wanda cnent strategies. 

An efficient performance analysis simulator generally 
simulates only those events that change the system state. Such 
simulators, called discrete event simulators, jump from event 
to event in simulated time. The length of the jumps (ie. the 
real time that would elapse between two events) does not 
effect the processing required to do a simulation; rather the 
total number of events is the key factor. Clearly the 
efficiency of the simulator is directly related to the level 
of detail of the simulation model. Increased detail is bought 
at the cost of more events being generated during simulation, 
and concequently more simulation processing. — | 

-MacDougall’s BASYS simulator [19] for a disk-based 
multiprogrammed computer system is a pr ime example of a 
discrete event simulator. It does stochastic simulations and 


is based on a queuing model. Events correspond to the 
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assignment of a job to the CPU, the release. of the CPU to wait 
for the completion of an I/O request, etc. Handling of the 
events is implemented in the simulator as event routines. Each 
event routine performs the actions which correspond to the 
associated event and also predicts followup. events (ie. 
schedules event routine executions). Event handling is 
maintained by. the simulator through the use of an event list 
(that reflects the time sequence of events yet to occur), 
queue information structures, and a.job table. Though the 
BASYS simulator is used to simulate a simple system, its 
Operating principles can be adapted eB do. more complex 
simulations. 

The character of the user-simulator interface. often plays 
a large part in determining the effectiveness of a simulator. 
Given an efficient simulator and an accurate simulation model, 
the user should be able to easily set up, run, and get back 
the results of simulation experiments. This dictates a 
flexible interactive environment, not batch processing. The 
Computer Networks Simulation System . developed. at the 
University of Waterloo [14] provides an example of a 
comfortable user-simulator interface, Network topology and 
traffic characteristics are input via a conversational dialog. 
Simulation output is available with or without data analysis. 
In addition, message delay information can be displayed during 


the simulation run. 
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2.3 Computer Network Simulation System Examples 


One way to review computer network simulation systems is in 
terms of the degree to which network components are 
represented in the corresponding simulation model. A 
simulation model may focus on a particuktar component of a 
computer network (eg. the communications links) and represent 
the rest of the network system at a very high level. A 
simulation system that incorporates such a simulation model is 
used to examine component behavior as opposed to overall 
network behvior. A simulation system that is used to examine 
overall network behavior requires a simulation model that 
represents all network components at some non-trivial level. 
An important simulation system design decision is to decide 
how completely the system is to be simulated. 

Many simulation systems have been developed that focus on 
the communications subnetwork in a packet switching network 
(7,12,13,26]. The communications subnetwork includes a number 
of dedicated communications processors interconnected by 
communications links. Each communications processor receives 
messages from one or more attached host devices (computers, 
terminal concentrators, etc.), routes messages through the 
communications subnetwork, and delivers messages to attached 
host devices when appropriate. The simulation systems that 
simulate this activity use simulation models that range in 


detail in representing the structures (eg. communications 
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links, communications processors) of the communications 
_ Subnetwork. The contrast between several simulation models is 
shown in Figure 2.1. 

The simulation model used at the University of Waterloo 
to study CIGALE (the packet=switching communications 
“subnetwork of the CYCLADES computer network) is illustrated in 
Figure 2.la. The objective of this study was to observe the 
behavior of CIGALE under various traffic conditions. To do 
thie line speeds, line delays and buffer utilization were 
accurately modelled. The lines were dseuned to be error free 
and the communications processor service time (to do packet 
routing and buffer handling) was modelled as a constant. These 
simplifications give an indication of how insignificant detail 
(in light of the modeller’s objectives) is kept out.of the. 
simulation model representation. | . | 

Figure 2.1b shows a simulation model, used at the Ecole 
Polytechnique in Montreal [12], which can be considered an 
enhanced version of the Waterloo model. A more sophisticated 
host device-communications processor interface is included, 
representing message segmentation into packets (eg. mes sages 
are allowed that are larger than the maximum packet size) and 
mes sage reassembly at the destination. Also packet 
transmission acknowledgement and retransmission in dase of 
nonacknowledgement (eg. line errors or insufficient packet 
buffering at.the destination) is represented in the simulation 


model. Overall, this simulation model can be used by a 
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Model of a Packet Switch (Water1lao) 
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Simulation system to generate more detailed packet (message) 
traffic information than the Waterloo system. 

A significant variation of the previously discussed 
Simulation models is shown in Figure 2.lc. This simulation 
model, developed at the Network Analysis Corporation [7], is 
intended to be used in simulations of communications 
processors in general. The interesting feature of this 
simulation model is the detail used to represent packet 
handling (routing, buffer management). This detail can be 
contrasted with the constant service time for packet handling 
in the Waterloo model. In focusing attention on the operations 
of the communications processor as opposed to network message 
traffic, a detailed computer system representation for the 
communications processor was required, 

A fundamentally different computer network simulation 
system was developed by Linsenmayer and Ligomenides [17]. 
Their simulation model combines a communications subnetwork 
model (such as those in Figure 2.1) and models of the computer 
systems attached to the communications subnetwork. In 
representing in some detail all major components in a computer 
network, the Linsenmayer model can be used to _ simulate 
computer networks as interdependent combinations of 
hardware/software elements and user job environments. In 
particular, studies are planned to examine various aspects of 
global job allocation in a computer network. The ultimate 


justification of a complete network simulation model, in light 
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of its inefficiency relative to limited models (eg. those in 
Figure 2.1), is its usefullness in generating overall network 
behavior. 

MNSS is used for studying the overall behavior of a class 
of network systems. As such, it is closest in design 
(functions and limitations) to the Linsenmayer and Ligomenides 


system. 
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CHAPTER 3 
DEVELOPMENT OF A SIMULATION. :SY¥YSTEM 


The development of another einviation system is justified 
| in light of the inadequacies of existing simulation systems in 
‘generating the behavior of constantly evolving network 
‘structures. Even where simulation systems accurately janeratce 
network behavior, there are often Limitations in the 
user-simulation system interface. The success | of MNSS| as a 
network avater analysis tool rests on its capability to 
provide a friendly user environment from which interesting 
network behavior can be studied. Each step in the MNSS design 
process was directed at achieving these goals. 

This chapter outlines the development of MNSS and 
identifies those features that make the effort worthwhile. 
The primary reasons for MNSS deve lopment will be ppeuented: 
followed by a description of the network system to be studied 
and ae general overview of the design of the corresponding 


simulation system. 
3.1 Motivation For The Development Of MNSS 

The motivation for the development of MNSS is based on 
two primary desires: the desire to study the beWavior of HPC 
network systems and the desire to build a. ‘better’ simulation 


system. 
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The design of a computer system (composed of one or more 
computers) is based to some extent on the anticipated behavior 
of the system in selected environments. The system design may 
be optimized fox throughput, response time, security or some 
special requirement. In all but ‘simple “environments, the 
behav ior ef most computer systems - is too complex to be derived. 
by intuition and mental wiearery. HPC nétwork systems are 
under development and there is a need for an analysis tool to 
study the comptes network behavior generated by alternative 
designs. In the initial Stages of devaiooaent sone of the 
areas of interest are protocol denign and resource management. 
Once network structures have been developed, the performance 
of HPC networks of various configurations under realistic user 
loads has to be examined. This must be done to perenne 
customize HPC networks to user specif ications, 

Simulation was chosen as the system analysis technique 
because of its capabilities. in generating behavioral 
information of complex systems at various stages in system 
development. The fundamental choice of simulation over 
measurement was based on practical considerations. while 
measurement is more accurate than sisuiation in deriving 
behavioral information, it cannot be used during early 
development when HPC networks ag not enist in aeasuceabie 
form, In addition, there are non-trival logistical problems in 
trying to measure the ireoreatce of large: network systems 


that operate with a variety of configurations and user loads. 
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The advantage of simulation versus analytical modelling is 
based on the limitations of mathematical models of complex 
sytems. A workable mathematical model of an HPC network system 
could not be developed to derive the scope of behavioral 
information that is required. In general, simulation is 
flexible, accurate, and efficient enough in generating HPC 
network system behavior, to be effective as an analysis tool. 

As was indicated in Chapter 2 many simulation systems 
have been developed, with a wide range of capabilities, for 
the study of computer network systems. Despite this abundance, 
there do not seem to be any available simulation systems that 
could be used to analyze HPC network systems. This is due, 
in part, to the uniqueness of the HPC network architecture 
(described in Section 3.2.1). Another reason to develop a new 
simulation system is the inpracticality of adapting an 
existing simulation system to a new operating environment 
(potentially different programming languages, processing 
capabilities, and i/o facilities). A third compelling reason 
to develop MNSS is the need for a simulation system that is 
useable in a commercial environment. There is a need for a 
better simulation system than has been previously offered. 

A ‘better’ simulation system is one that is developed 
according to the principles reviewed in Chapter 2. Using those 
principles a simulation system can be made easy to use, 
flexible enough to perform a wide variety of simulation 


experiments, accurate in generating network system behavior, 


30 


and cost/feffective as an analysis tool. Any one of these 
features can be found in currently available simulation 
systems, but few systems integrate them all into an effective 
package. A gaal of the MNSS development effort is to provide 
an integrated system in which all facets of the system reflect 
general design goals of flexibility, accuracy, ease of use, 
and efficiency. This applies to the simulation model, 


simulator, and user interface. 
3.2 Development Of MNSS 


MNSS was developed to incorporate features that are 
advantageous for a general purpose simulation system. The 
modelling discipline, simulator, and eer interface are not 
fundamentally restricted to the simulation of any one class of 
computer systems. The primary aaae of MNSS_ though, is to 
simulate the behavior of HPC network systems. The HPC network 
architecture that is particularly suited to the formation of 
minicomputer networks. Therekere: “Minicomputer”’ Network 
Simulation System (MNSS) is an appropriate designation for the 
simulation system described in this thesis report. The 
following two sections present the general characteristics of 
the HPC network “architecture and an overview of the MNSS 


implementation and use. 


31 


3.2.1 The HPC Network Architecture 


There are two basic aspects to the ‘HPC network 
architecture: 1. the architecture of the individual “HPC 
systems, and 2. the way in which Wetwork ‘communications is 
handled by an HPC system. ae 

An HPC system BURPOERS a nul tiprogramning environment in 
which _ several independent programs canbe ‘executed 
concurrently. A program is made up of one or more processes; a 
process is composed of the software “control structure for a 
particular execution of code. code ane data: may be shared 
between processes within a program. all contention ‘for ayeter 
resources (eg. CPU, menory; etc.) occurs at the Srocess level. 
The process control software uses a priority-ordered 
preemption | scheme to arbitrate ‘resource | contention. 
Time-slicing is used _ to delegate usage of resources such as 
the CPU among ‘processes of equal prior ity. 

There are two basic types of processes: u user processes 
and system BRORCR ECE: User procerree are the principal agents 
of a user in getting work done on an HPC system. The priority 
level of user processes may vary, but typically it is fixed at 
a particular level. User processes can be associated with 
several forms | of activity, Sscisdiag: code execution 
(processing), short wait operations, long wait operations, and 
remote processing operations. _ The distinction be tween SROEE 


wait operations (eg. disk i/o operations) and “long wait 
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ee 


operations (eg. terminal i/o) is made Becaune long waits force 
a process to lose control of copresked resources such as the 
CPU, whereas short waits do not. ‘Remote processing operations 
relate to remote process activity ‘that is initiated by a 
remote request. A ‘Peeves will make a ) remote request in order 
to do work on a fenore system in | an HPC petucrk... This may 
include accessing a network database or running a program that 
works only on a PSEtrcuree system | in a network. A user process 
‘waits’ for a remote eave to conplete before it resumes 
local activities ie: processing, wait activities). 

A system process is. generated by the HPC operating system 
to support user acitvities. “Through systen processes, users 
can utilize HPC hardware and software ‘resources ‘that cannot 
be accessed by user processes directly. the reasons for ‘using 
system process intermediaries are to maintain the independent 
activities of concurrently existing processes, to. act as 
agents for user process communication activities, gaa to 
relieve users of the complications of low level HPC 
operations. System ‘eceewaee are at higher ericeity levels 
than user processes and eorsaguenely: preempt ‘user processes in 
cases of resource contention. _ | ; 

In a network environment the systen processes most often 
seen are agboctatec with commun ications activities. These 
system processes | handle Somuuhi cations request/response 
initiation and network atore>and=-forwacd operations. Once 


communications system processes gain CPU control (by waiting: 
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in line behind other system processes or by preempting a user 
process), they do processing, short waits, and link control 
operations to perform a communications function. There is no 
long wait activity and, due to the brief time it takes to 
perform a communications function, the communications system 
process is not preempted (by another system process) from CPU 
control. 

HPC networks can be arbitrarily connected in locally or 
geographically distributed configurations. Store-and-forward 
facilities are _used whenever information must be transferred 
between HPC systems that are not directly connected. The 
communications links in an HPC network may be half or full 
duplex, and can transfer information ata variety of baud 
rates (1200, 2400, ...). System processes maintain the 
protocols (eg. SDLC, X.25) that are used to control the links. 

There are three types of information transferred through 
an HPC network: request messages, response messages, and. link 
control messages. Request and response messages are generated 
by a system process for a user process. The transmission and 
routing of these messages is also handled by a system process. 
A request message is generated when a user process seeks to do 
work on a remote HPC. system. When it arrives at the 
appropriate destination, a user process is started to do the 
requested work. When the work is completed, a response message 
is generated. This message is then sent back to restart the 


waiting initiator user process. Link control messages are used 
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‘primarily to acknowledge the transmission of request/response 
messages and to aid recovery from link errors. 

All communications overhead (eg. use of CPU and memory 
resources) is handled by the “host” systéms in an HPC network; 
there are no communications front end processors or network 
store-and-forward machines. System processes, which share 
resources with user processes, are used to do network 
communications. The performance of an HPC system is degraded 
for  lacal activities in order to provide network 
communications. capabilities. This is a reasonable tradeoff in 
a minicomputer network where an investment in dedicated 
communications hardware is not cost-effective. The HPC nétwork 
architecture is, therefore, primarily an architecture for 
providing low cost communications services for a network 


composed of minicomputers. 
3.2.2 MNSS Design And Use 


The two key steps in the MNSS development effort were, 
first to formulate flexible models to represent HPC network 
systems, and then to. build a simulation system using these 
models. The guidelines for the development effort were 
dictated by the goals set forth in Section 3.1, and reflected 
many of the principles discussed in Chapter 2.° 

MNSS models, which are representations of HPC network 


systems, are built up from a basic model of an HPC system. 
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Inherent in the HPC system model is an _ interactive 
relationship of the representations of processing and 
communications mechanisms. This relationship is essential to 
the HPC network architecture. Overall, MNSS models can be 
characterized as complete network models. This is true because 
all major network components (ie. local processing and network 
commun ications) and their interactions are represented 
non-trivally. MNSS models are described in a functional 
modelling language that has a small number of building blocks 
to represent HPC network system features. 

MNSS is implementated on an HP3000 computer in SPL, an 
Algol-like programming language. MNSS incorporates a 
discrete-event simulator and an interactive interface. The 
simulator uses an SPL version of an MNSS model to generate HPC 
network behavior. The MNSS user interface is implemented as an 
interactive dialog. In many cases the user picks from a menu 
of alternatives in order to run a simulation experiment and 
get back results. | 

There is a wide range of behavioral information that can 
be generated by the simulator (due to a large extent to the 
completeness of MNSS models). The user can request’ the 


following results to be displayed by MNSS: 
1. processing statistics broken down by node and process 
type (eg. system process, locally or remotely initiated 


system processes), 
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2. 


4. 
5. 


7. 


cumulative remote processing initiated by local 


processes at each node, 


queuing statistics (eg. maximum length, mean length, 


maximum wait, mean wait) for user processes in the CPU 
wait queue, . | . 

queuing statistics for messages in the link wait queue, 
CPU utilization statistics broken down by node and 
proceuest state (eg. system or user process control, 
idle, process handling overhead) , | 

link utilization statistics broken down by link state 
(eg. idle, transmitting), 


the number of user process launchings (intiate CPU 


_ control) per hode, and 


This 


the number and average size of message transmissions per 
1 ink e 


information gives a user the ability to do sophisticated 


studies of HPC network systems. 
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Chapter 4 
MNSS MODELLING 


The simulation of a system initially requires that a 
model of the system be conceived. MNSS utilizes a set of 
related models that are abstract representations of HPC 
network systems. The development of ‘these models (refered to 
here as ‘’MNSS models’) was based on two general design 
criteria. First, the models had to be applicable for use ina 
simulation system. Essentially the cost, in terms of human and 
processing resources, to develop the simulation system and to 
run simulation experiments had to be minimized. This required 
reasonably high level models of HPC network systems and a 
straightforward procedure for the implementation | of these 
models in a simulation framework. The second design criterion 
was that the models be made to contain the variables and 
relations that are significant in representing the behavior of 
HPC network systems. This design criterion was balanced 
against the first design criterion, insuring that important 
detail was not purged from the model in order to simplify 
simulation system development and use. 

The effort to design and implement MNSS models ran within 
the framework established by the design criteria. The first 
step was to develop a modelling discipline. This involved the 
development of a modelling language and a technique to 


conveniently map ‘real life’ system structures into the 
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abstractions of the modelling language. The next step was to 
construct high level building blocks from modelling language 
structures that could then be used to build MNSS models. The 
structural similarities of HPC networks facilitates the use of 
these building blocks; this avoids starting from scratch in 
modelling each different network. The final step in the 
modelling effort was to develop a systematic procedure to map 
modelling structures into structures in the simulation system 


programming language. 
4.1 The MNSS Modelling Discipline 


The MNSS modelling discipline facilitates the development 
of functional models of systems. It represents a 
generalization of techniques used in finite-state modelling 


and queueing network modelling. MNSS modelling structures have 


- close analogs in the structures of these modelling 


disciplines. The five basic MNSS model structures are 
entities, groups, group transitions, entity sources and entity 
sinks. | | 

An ‘entity’, in the context of the MNSS modelling 
discipline, is an object whose behavior is of interest. It 
could be a shopper in a supermarket model Or a query ina data 
base model (note that the MNSS modelling discipline is general 
enough to be used to describe models of svekens other than HPC 


computer networks). Associated with an entity is information 
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about those entity characteristics that have a bearing on its 
behavior in the system to be modelled. Entities can be 
classified according to common characteristics into ‘entity 
classes’. Entity classes are used as a convenient way to deal 
with entities (particularly the information associated with 
entities) durning the construction of a model. 

A “group” is a collection of entities showing a 
particular form of behavior; the behavior of an entity over 
time can be described by the sequence of groups it was in. A 
group can incorporate a variety of orderings on the entities 
(of one Or more entity classes) that occupy it at any point in 
time, ranging from no Siaevind to a sophisticated queue 
ordering (eg. FIFO, according to entity class, based on 
priority information associated with an entity, etc. ). For 
the supermarket model a group could be defined for the 
shoppers waiting to be checked out or the shoppers be ing 
checked out. The waiting line (a FIFO queue) is the ordering 
in the ‘waiting’ group. 

Entities can come into existence in a mode 1 of a system 
in a number of ways. They may be defined to exist in a 
particular group when the model is initialized (in general as 
permanent entities ina closed system model); alternatively an 
entity may be created and injected into a group in the model 
from an ‘entity source’. Associated with an entity source is 
information that identifies the class of entity to be created; 


this information that is used to generate entity 
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characteristics. An entity can cease to exist in a model by 
being absorbed by an ‘entity sink’. By using entity sources 
and entity sinks, models of open systems ( ie. systems that 
interact with their environment ) can be formulated. 

A “group transition” is a path from one group to another 
group, from an entity source to a group, or from a group to an 
entity sink. Each group transition has associated with it an 
event routine. The event routine identifies the consequences 
of an entity leaving a group or entity source via the group 
transition. The consequences of a group transition may include 
provocation of other entities to undergo group transitions 
(immediately or at some future time), changes to group entity 
orderings, or modifications of entity information. For 
example, in the supermarket model a shopper leaving the “being 
checked out’ group will cause: another shopper to do a group 
transition to that group from the ‘waiting’ group, a group 
transition to be scheduled for the new occupant of the ‘being 
checked out” group (when all that shopper’s groceries have 
been checked out), and the ‘waiting’ group to be reordered. 

In order to provide a convenient mapping between a ‘real 
life’ system and a model of that system, a symbolic notation 
has been defined for the MNSS modelling structures; this 
notation is illustrated in Figure 4.1. The presence of an 
entity class symbol within the symbol for a group, entity 
source or entity sink indicates that entities of that 


particular class can occupy the group, source or sink. One or 
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( ) entity class "Eg"! 


8 group ( with entity classes E or Ey ... En ) 


Cran entity source 


oe entity sink 


———_——_———-> group transition 


Figure 4.1  MNSS Modelling Structures 
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more entity classes can be associated with a group, source or 


sink. 
Figure 


rules: 


l. 


4. 


A 


shown 


MNSS modelling structures (and the symbols shown in 


4.1) are connected together according to the following 


Groups can be connected together by a group transition 
as long as they have at least one entity class in 
COMMON | 

An entity source and a group: can be connected by a 
group transition, leading from the source to the 
group, as long as they have at least one entity class 
in common; 

A group and an entity sink can be connected by a group 
transition, leading from the group to the sink, as 
long ae tney have at least one entity aes in comon; 
One or more transitions can lead to and from a group, 
from an entity source, or to an entity sink; 

A path in a model established by the connection of 
groups associated with a particular entity class must 
be either a closed path or a path from an entity 


source to an entity sink. 


job processing model created using the MNSS notation is 


in Figure 4.2. A job (an entity in the model) comes 


from an external job source and goes into a FIFO queue ordered 


processing-wait group. A job is transferred to the processing 
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2. 


job introduction 
() Jobs 


job processing begins 
job processing suspended 


job processing completes 


Figure 4.2 A Job Processing Model 


hy 


group when it is at the head of the FIFO queue and the 
processing group is uneccupied by any other job. A job stays 
in the processing group until it either completes the 
processing it has to do, in which case it is absorbed by the 
job sink, or it uses a system allocated amount of processing 
time (ie. occupies the processing group for that amount of 
time), in which case it returns to the processing-wait group. 
This model is very simple and not very useful (it assumes 
among other things that there is Ho overhead associated with 
moving a job in and out of the processing group). In erder to 
get a more detailed model, additional groups could be added, 
more descriptive information could be associated with jobs 
(ie. the amount of memory required), and the group transition 
event routines could be made more scgkisticated (eg. taking 
into accennt job memory cequitements, 4op handling overhead, 
etc.). a 

In general the following procedure “should be used in 


modelling a system: 


1. Isolate the system of interest and identify all 
interactions between the system and its environment; 

2. Identify those objects in the avatee whose behavior is 
of interest and set up entity classes (with entity 
characteristic information) ; 

3. For each entity class identify the types of behavior 


that are possible and establish a network of groups 
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for the aignificany aepects of this behavior; 

4. Consolidate the groups that are essentially | the same 
but that have different entsey classes; 

5 Def ine group transition event “routines for each oF OUP 


transition in the model. 
4.2 The Structure Of MNSS Models 


MNSS models are Resigned to accomodate the behavior of 
user processes, system processes and messages in HPC ‘network 
avetens. The composi te behavior of these objects Sencompasees 
all interesting system behavior as identifies in Chapter 3. 
Accordingly user BEOCESS). system process and message entity 
classes have been def ined. For an: MNSS model, entities from 
these classes exist in a network of groups, entity, sources and 
. entity sinks structices to represent all significant entity 
behavior. | . | | 
| MNSS models can be built waing an BPC system model as a 
basic component. As such, the HPC system model has to be 
adaptable to arbitrary communications ZOone Sure Crone (ie. 
network environments). A submodel structure has been seceiated 
to enhance the STORIE EE EEY of the Brite byeter model. The 
submode 1s that have been constructed are essentially another 
(lower) level of MNSS model components. 

Two submodels make up an HPC system model; they are the 


processing submodel and the. comnun icat ions submodel. “The 


46 


processing submodel represents vie behavior of user and system 
sidcedeaes ' The behavior. of these processes is affected by, 
and affects, the flow of messages in the Communications 
submodel. the interactions batinen the two submodels emanate — 
from each submodel’s group transition event routines. There 
are no group transitions between the processing submodel and 
the communications submodel, but a group transition in one 
submodel may trigger (by way of an avant routine) a group 
transition in the other. | aan y 
| To form ‘ complete MNSS most: the communications 
submodels of several ‘HPC ayste™ models are connected together 
by group ceaweittons: If two ‘HPC systens are tied together by 
a communications Link in the network to be modelled, then the 
corresponding communications submodels are connected. 

The overall structure of “an | HPC systen model mode 1 is 
shown in Figure i oe The details of submodel organization and 
interaction. are discussed in the following sections on the 


processing submodel and the communications submodel. 
4.2.1 The Processing Submodel 


The processing submodel is illustrated in Figure 4.4. 
This submodel consists of four groups, two entity sources, two 
entity sinks and eight group transitions. These structures 
are arranged to provide a representation of an HPC system 


which describes the significant behavior of user and system 
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Figure 4.3 General HPC System Model Structure 


48 


user process 
source 


user process 
sink 


CPU-wait 
group system process 
system process sink 


source 


Figure 4.4 Processing Submodel 
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processes. The entities that flow through the submodel are, 
naturally eiadah: user processes and system processes. Each. 
process entity has information associated with it which, along 
with group ordering and transition infovmationy determines 
when group transitions ste to occur. The watdre of the 
information associated with a particular procasa entity is 
based on it’s entity class. | | | : 

The following items of information are associated with 


each user process entity: 


1. The ‘user process type’ identifies the user process as 

| being either locally or remotely initiated. 

2. The ‘complete time’ is ‘the ‘processing time (ie. 
residence in the CPU group) required for user process 


completion. 


3. The “long wait time’ is the processing time until the 
user process initiates a long wait or, if the user 
process is doing a long wait, the time when the long 
wait soap letes. 

4. The ‘remote wait time’ is the processing time until 
the user process initiates e. emote request (fora 
locally initiated process) or a completion response 
(for a remotely initiated process). For a remotely 
initiated process this item aie “equivalent to the 
complete time. | 


5. The “process time distributions’ are the time 


50 


6. 


7. 


8. 


9. 


distributions from which information items 3 and 4 are 
generated (eg. when a user process terminates a long 
wait the processing time until the next long wait is 
generated using the appropriate process time 
distribution). | | | 

The ears requirement’ specifies the amount of 
primary memory needed to hold the user process code 


and data (ie. the information that needs to be brought 


_ into primary memory to allow the process to run). 


The ‘remote destination distribution’ is the 
distribution that is used to generate the destination 
of a remote request for a locally initiated user 
process, or the response destination for a remotely 


initiated user process (in the latter case the 


distribution deterministically generates the source of 


the remote request). 

The ‘message size distribution’ is used to generate 
the size of a remote request message or a completion 
response message. 

The ‘remote process distributions* are defined for 
locally initiated processes only. They are used in the 
generation of a remote user process that is initiated 
as the result of a remote request. These distributions 
include the process time distributions and the message 
size distribution (for the reply generated by the 


remote process). 


51 


A aveeen process exhibits much simpler behavior than a 
_ user process; a eyeten process does not do long waits, and 
also does not initiate remote requests. Consequently, the 
information associated with a system process entity is a 
subset of that associated with a user process entity. The 
following information items are defined for each system 


process entity: 


1. The “system process type identifies the task of the 
system process as either initiating a “remote request 
communication or handling a message from a remote 
source (ie. routing it pehrough ‘the communications 
system). _ 

2. The ‘complete time’ is the processing time necessary 


to complete the appropriate system process asks 


The four groups in the processing submodel are designated 
the CPU group, the CPU-wait ereup; the long-wait group, and 
the remote-wait group. The cpu group. can be occupied by a 
single entity from the system process entity etass or the user 
process entity class. A pECcess: in the CPU group pee control 
of the HPC CPU. The process may be doing either processing Or 
a short wait, which does not result inca “switch of CPU control 
to another process (as discussed in Section 3.2. 1). When the 
process is preempted from CPU control, completes, initiates a 


long wait, or initiates a remote request, an appropriate group 
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transition from the CPU group will occur. Preemption will 
cause an entity transition to the CPU-wait group; a long wait 
initiation results in a transition to the long-wait group; the 
initiation of a remote request by a process brings about a 
évansition to the remote-wait group; _ and when a process 
completes it is swept away to an entity sink (user processes 
to the user process sink, system processes to the system 
process sink). 

The three ‘wait’ groups characterize the behavior of a 
process entity when it is not in control of the CPU. If a 
process entity is in the CPU-wait group then the process is 
waiting to gain control of the CPU. The CPU-wait group has a 
dual FIFO queue ordering for system and user processes. 
System processes and user processes gee kept in seperate 
queues. When the CPU group is unoccupied and the CPU-wait 
group is occupied then a process will make a group transition 
from the CPU-wait group to the CPU group. The process that 
Makes the transition is taken from the system queue if it is 
‘nonempty, else it is taken from the user queue. The presence 
of a system process in the CPU-wait group causes a user 
process in the CPU group to be preempted. The user process 
undergoes a group transition to the CPU-wait group and is 
positioned at the head of the user process queue, becoming the 
next user process to be served. In addition, a user process 
can be preempted from the CPU group if it has done a ‘user 


Slice’ of processing and there are other user processes in the 
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CPU-wait group. In this case the preempted user process is 
placed at the tail end of the user process queue in the 
CPU-wait group. 

The long-wait group and the remote-wait group can only be 
occupied by user process entities (system process entities do 
not exhibit the behavior these groups’ represent). The 
long-wait group is occupied by a user process entity when the 
entity is doing a long wait. This group has an ordering of 
user processes’ based on the length of time each process must 
remain in the group. When the long wait of a process has 
completed the process undergos a group transition to the 
CPU-wait group and is placed in the user process queue. The 
remote-wait group is occupied by a user process when the user 
process has initiated a remote request. The remote-wait group 
does not impose an ordering on the entities that occupy it. 
When a remote request has completed (ie. a response message is 
received at the HPC node), the user process that initiated the 
request is removed from the Lamereowalt.6roup: and placed in 
the CPU-wait group. 

The process entities that flow through the processing 
submodel can be generated at MNSS_ model initiation to 
represent a permanent user load, or they can be generated 
dynamically to represent a user load that changes over time. 
The processes generated as part of a permanent user load are 
always user processes and are placed initially in the CPU-wait 


group (when the model is used for simulation, care should be 
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taken to avoid biasing CPU-wait queue statistics as a result 
of this initial condition) ; system processes are generated as 
a result of the actions of user processes and as such cannot 
be in existence at model initiation when no action has taken 
place. Processes are generated dynamically at the system 
process source and the user process source, and are injected 
(via group transitions) into the CPU-wait group. “The user 
process source generates user processes with all the 
associated information (complete time, etc.). The user 
process generation capacity of the user process source, 
coupled with the group transition event routine (for the 
transition from the user process source to the CPU-wait group) 
that dictates the frequency of entity generation, can 
completely represent a dynamic user load on an HPC system. 
The user process source also generates user processes to 
satisfy remote requests. The ieaneition of this type of user 
process into the CPU-wait group is. triggered by the completion . 
of the system process that handled the incoming remote 
request. | . 

The system process source generates system processes to 
handle communications processing. The processing may be 
required for the initiation of a remote request or for 
handling the arrival of a response. A veucte request is made 
by a locally initiated user process entering the remote-wait 
group. The completion of a remotely tareieted process (ie. the 


process is absorbed by the user process sink) signals the 
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initiation. of a remote response. Both the Eencrs request and 
response cause identical communications processing and hence 
the same type of system process is generated (le. with the 
same completion times). A dif ferent type of system process is 
, generated by the system process source to handle the 
\eaunieations processing for network message routing. This 
processing is triggered by a group transition in the 
communications submodel (to be discussed in the next section 
“in detail along with other interactions between the processing 


_submodel and the Gémmunications submode}) . 
4.2.2 The Communications Submodel 


The communications submodel provides a representation of 
the important behavior of the communications message flow. 
‘This submodel is flexible enough to be adapted to a wide range 
of communications configurations. Two variations on the 
communications submodel are shown in Figure 4.5. This Figure 
illustrates how half and full duplex communications 
capabilities are integrated into the submodel. In generat the 
submodel incorporates three types of groups: (two of which are 
“duplicated for each communications Link modelled), an entity 
source, an entity sink, and five principle group transitions. 
The entities that flow through the submodel be long to ene 
message entity class. 


The information associated with a message entity controls 
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Figure 4.5 Communications Submodel 
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the message flow through the communications submodel. This 
information is used by group trangition event routines to. 
direct the message: to its local Ge“ pamote destination. In 
addition, once a message reaches its destination the entity 
information is used to determine the correct group transition 
action to be taken (eg. restart a local user process or 
generate a remotelly initiated user process). The message 


entity information includes the following items: 


1. The ‘message type’ classifies the message as either a 
remote request message or a response message. - 

2. The ‘message source’ identifies the user process which 
initiated the remote eoadest ana Caused the message to 
be generated (for a response message ‘the message 
source is the user process that originally initiated 
the remote request). | . 

3. The ‘message destination’ is an identification of the 
HPC syaten to which the message is directed. 

4. The ‘message length’ is the length of the message in 
bytes. ‘ | 


The three types of groups in the communications submode 1 
are designated the message-in group, the link-wait group and 
the link group. There is. only one mes sage-in group per 
conmanications submodel, but there can be duplicates of the 


link-wait group and the link group. Essentially the “link 
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section’, blocked off in the submodel illustrations of 
Figure 4.5, is duplicated for each commun ications link (half 
or full duplex) attached to the HPC system being modelled. The 
flexible structure of the commun cartons submodel iends itself 
to representation of a wide variety of HPC system 
communications configurations. | 

The message-in group can be occupied by multiple mes sages 
with no ordering enforced on the messages. Messages occupy the 
message-in group when they have been received by the HPC 
system and are being processed. This means that a for each 
message in the sessage-in group a system process, generated 
when the message entered the message-in group, is ative in 
the processing submodel. The system processing to be done 
upon arrival of a message into the message-in group is for 
message routing activities. This includes determining the 
message destination; if the destination is that particular HPC 
system then the appropriate user processing novia is taken,. 
else the message a started on. the’ way. to its remote 
deat ination: 

The action taken on a message, and the accompanying group 
transition from the message-in group, occurs when the 
corresponding ‘system process in che ‘processing submodel 
completes and is absorbed by the system process eink. If the 
message is directed to a remote destination, the message will 
make a group transition to an appropriate Link-wait group. The 


particular link-wait group chosen is determined by a routing 
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algorithm incorporated into the group transition event routine 
(the link-wait group is selected such that the message will be 
routed correctly to its destination over the link associated 
with the link-wait group). If the message is directed to the 
local HPC system then a transition will occur to the message 
sink where the message will be absorbed. If the message is a 
response to a remote request, this transition will trigger a 
transition in the processing _ submadel from. the remote-wait 
group to the CPU-wait group for the user. process that 
originally initiated the remote request. — Otherwise the 
mes sage igs a remote vequant ang_a remqtely initiated user 
process will be generated by the user process source and 
injected into the cpU-wait group. 

A message will be launched from the message source to a 
link-wait group when the system process initiated to handle a 
remote request completes (ie. the process is absorbed by the 
system sink). The message generated by the message source will 
be assigned descriptive information (source, destination, 
length) based on information associated with the user process 
making the remote request. . 

In general the interactions between the processing 
submodel and the communications submodel occur in conjunction 
with transitions by messages to and from the message-in group 
and transitions from the message source. These interactions, 
which are designed into the event routines, are the key to 


modelling communications overhead in the HPC ..system. 
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The key to modelling a variety of communications 
configurations is the flexibility to customize the 
organization of the link and link-wait groups in the 
communications model. 

The link group can be occupied at any time by at most a 
single message. A message in the link group is in the process 
of being transmitted. (physically) from one HPC system to 
another. Consequently each link group is part of two 
communications submodels; it is the br idge that ties together 
the models of individual HPC systems to form MNSS models. A 
‘message leaves the link group and enters the message-in group 
in the communications submodel of the HPC system to which it 
was transmitted when the transmission has completed. The time 
of transmissiion is based on the length of the message (a 
message entity information item) and the speed of the link 
(taken into account in the link to message-in group transition 
event routines). Corresponding to every link. group in the 
communications submodel there is a link-wait group. Messages 
in the link-wait group are ordered in a FIFO queue. When the 
link group becomes free for entry of a message (ie. there is 
no message in it and the delays due to physical link control 
have elapsed) the head message in the link-wait message queve 
makes the transiton to the link group. 

The event routine that is associated with the group 
transition from the link-wait group te the link group models 


the link control protocol used on the corresponding link in 
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-the real network system. For every variety of control 
protocol there is a distinct event routine, For example, a 
half duplex link sonereR event routine would be concerned with 
delays in turning the link around and in switching a link from 
an idle toa transmit-ready disposition. On the other hand, a 
“hard-wired full duplex link would have an event routine that 
is only concerned with whether the link ‘¢ free or not (the 
link is always tranenit-teady and oriented in “the right 
direction). . | | 

Once the appropriate event routine for a link control | 
protocol has been defined, the next step in mousing a link 
in the communications submodel is to make the appropriate link 
and link-wait group connections. The differences in the 
conections for a half and full duplex link can be seen in 


Figure 4.5. 
4.3 Simulation Using The HPC System Model 


Several features of the HPC system model enhance its 
potential as a simulation model. Among these are the types of 
structures used in the model, the model’s levels of detail in 
representing significant HPC behavior, and the flexibility of | 
the model as a building block for MNSS models. 

The structures defined by the MNSS modelling discipline 
and ~ used “in the HPC system mode 1 can be _ conveniently 


manipulated in the context of a simulation system. The 
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information associated with MNSS node 11 ing structures can be 
implemented as data structures ‘an the ‘simulation programming 
language. For example, each entity class can have an 
information table, with each entry in the table cor responding 
to an active entity in the ‘model. The group ‘transition event 
routines that control the action in the model -can be 
implemented as EEconenree ‘(pieces of code that are callable on 
demand by the. hain simulator program). ? A group transition 
occuring in the simulation model then results in execution of 
the appropriate transition procedure. In general, MNSS mode 1 
structures can be systematically converted to the programming 
language structures of the simulation ‘system. _ 

Another factor that affects the conversion ‘of a model for 
use in simulation is the level to which the model represents 
a system’s behavior. A model can fail to be an effective 
simulation model if it represents a sretee with too much or 
too little detail. Too much detail (ie, the model represents 
some system behavior at too low a level}: ean make the model’s 
use ih simulation a torturous exercise in overkill. On the 
other hand, a model which does not represent certain aspects 
of a system in enough detail would not be of use in simulating 
all the significant system behavior. | 

The HPC system model utilizes two primary levels of 
detail to produce a valid ana effective model of an HPC 
system. The processing submodel represents one level of 


abstraction of system structures. The communications submodel 
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represents structures at another, higher level. The message 
in the communications submodel is an entity that is created, 
Manipulated and destroyed by the processes of the processing 
submodel. . — 
| By modelling messages separately from processes, easy 
correspondences can be made pecveen HPC. network communications 
structures and structures in the HPC system model (eg. 
communications Links and _ protocols) . : This multilevel 
- modelling technique also avoids | the | complexity “hia 
inflexibility of representing communications in a Processing 
model (in particular the group transition event ‘routines yout? 
be very hard to def ine) . 3 
Perhaps the key feature of the HPC system model is the 
way it can be ised as a building block for -MNSS models. This 
capability was shown in Section 4.2.2 in the discuseton the 
use of the link group in connecting: HPC system models 
together. Basically a network of commun ications submodels is 
formed, with each communications submodel having a 
corresponding processing submodel. The simulation system 
implementation of this network mode 1 structure is facilitated 
by adding node and link sdavitiere to the MNSS model group 
names. These additional qualifications are necessary since 
there are several copies of each group in a network model made 
up of several submodels. With each group ine a network 
unambiguously identified, simulation system data structures 


can be easily implemented and managed. 
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CHAPTER 5 
MNSS IMPLEMENTATION AND USE 


There is a great deal of latitude in implementing MNSS. 
The details of a particular implementation ay be shaped by 
the specifiations of the couputed system on which MNSS is 
implemented, or by the needs of the network analyst who is to 
use MNSS. On the other hand there are a number of essential 
features that should be common to all ef fective 
implementations. These features involve the overall structure 
of an MNSS implementation, the representation of a simulation 
experiment, and the basic characteristics of the simulator. 
By nature they are both useful and feasible within the 
restrictions of any particular implementation. ‘Therefore an 
accounting of these features should be aade.to.quide MNSS 
implementors. | A 

The following sections provide a discussion of iapottant 
MNSS- implementation independent features. Appendix A 
supplements this discussion with a brief description of some 


interesting aspects of a particular MNSS implementation, 
5.1 Overall Structure Cf A MNSS Implementation 

The overall structure of a MNSS implementation should 
reflect an emphasis on top-down and modular organization. One 


possible structure that does this is modu lar ized based on MNSS 
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functions. A graphic view of thig structure is given in 
Figure 5.1. 


Five major MNSS functions have been identified: 


1. create a simulation experiment specification, 
2. recall a simulation experiment specification, 
3. modify a simulation dvparieant specification, 
4. save a simulation experiment specification, and 


5. run a simulation experiment. 


These functions represent effective tools for performing 
simulation experiments. A simulation experiment specification 
is created by first initializing all relevent information 
tables and then, through user. interaction, building a basic 
network specification (ie. the number of nodes in a network 
and how they are connected). Once an experiment specification 
has been created it can be modified interactively to produce a 
variety of related specifications. In particular network 
configuration iAeocuetion (es: Link speeds) and user load 
information can be manipulated. It is possible to create 
nonvolatile copies of an experiment specification, that is, 
copies that do not disappear when MNSS use is terminated; 
experiment specifications can be saved and recalled later when . 
they are needed. A user can choose to run a simulation 
experiment at any time with the available experiment 


specifications. 
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Each of the major MNSS functions can be implemented asa 
super module containing numerous subfunction modules (each of 
which. may contain | sub-subfunction modules, etc.). This 
modularization according to subfunction is shown in Figure 5.1 
for the major function “run experiment’. 7 - | 

At the top of the top-down organization shown in 
Figure 5.1 is a module which acts as a switch between the 
major functions. In particular this module is implemented to 
switch control, at the request of the user, between the five 
major functional modules; the only exception is when MNSS is 
first started up, in which case the user only has the freedom 
to create or recall an experiment apects cations Any othee 
function would be invalid because of the lack of an experiment 


specification to operate on. 
5.2 Representation Of A Simulation Experiment 


| The items of information eit are needed in the 
representation of a MNSS experiment are dictated by the 
characteristics of the HPC network models. For example, the 
network representation must include the number of nodes and 
links, the connectivity of the het work: and link deecriovive 
information (eg. half or full duplex, link speed, etc.). The 
Management of these information items is to a large extent 
MNSS-implementation dependent, but there are some information 


Management policies that are generally useful. Two such 
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policies deal with providing alternative representations for 
individual information items and minimization of the 
information redundancy in an experiment representation. 

In most cases a MNSS_ experiment representations will 
contain a large amount of redundant information if the user 
load for each node in the network is represented seperately. 
This redundancy often results from the use of a standard user 
load that is specified for more than one node in the network. 
To reauce redundancy, information can be kept in ‘common’ 
areas where it can be accessed by (ie. linked to) higher level 
information structures in the experiment representation, 
Information items or structures that are potentially used more 
than once in an experiment are candidates for ‘common’ status. 
This applies to everything from distribution specifications to 
complete process specifications. 

Figure 5.2 illustrates an experiment representation 
breakdown with common process’ characteristics and common 
‘distributions. The common process characteristics, that might 
be grouped together and shared by processes, include process 
time distributions, memory requirements and message size 
distributions. A separate table(s) could be used to store 
distribution specifications. Entries in this table(s) would be 
pointed at by entries in the process table and the common 
process characteristics table. 

The success. of the common information item representation 


scheme rests on three factors. One is the storage requirements 
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of a reference link relative to the requirements of the common 
information | item. If an information item needs more storage 
than a link to that item, then tne dntcarmation item may be 
best kept ina fomee table. The second factor is the number 
of references made to. an istocustion item. -in general, the 
smaller the storage difference between the item and a link to 
it, the larger the number of references that are needed to 
justify common status. The third factor for jsuccess is the 
specification time that can be saved. bya user in creating 
simulation experiments with standard process specifications. 
Modifications of existing simulation experiments een also be 
greatly simplified. For most foreseeable MNSS implementations 
a common information item Ssgunieation for experiment 
representations is attractive in terms of system efficiency 
and user convenience. 

Alternative representations for individual items can be 
provided in a MNSS implementation by allowing a user to select 
from a variety of distribution types. These types could 
include constant, special, uniform and exponential 
distributions. Figure 5.3 illustrates the ‘various distribution 
functions. | 

The constant, uniform and exponential distributions are 
commonly used in modelling and simulation, The function used 
for the exponential distribution is taken from MacDougall’s 
tutorial paper on simulation [19]. This function can be 


adapted to generate a finite range of values by specifying a 
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maximum and minimum value. Values generated outside this range 
are either discarded or replaced by the closest valid value. 
This modification of the exponential distribution function is 
useful because information items do not always range in value 
from zero to infinity (note that constant and. uniform 
distributions have limits). 

The special distribution is useful for information items 
with values that can only be generated from an unusual 
distribution (ie. not constant, uniform or exponential). A 
special distribution consists of a finite number of values, 
each of which has a probability. The sum of the value 
probabilities is one. The number of values in a special. 
distribution is dictated by the experiment specification 
Hequicenente of the paruicuiae simulation study in which the 
special distribution is used. In general ai special 
distribution can be used to approximate any distribution and 
is the alternative when constant, uniform or exponential 


distributions are inadequate. 
5.3  MNSS Simulator Characteristics 


An event-driven simulator is especially suited to the 
simulation of systems characterized using the MNSS modelling 
discipline. Events in the MNSS modelling discipline describe 
the effects of the state Changes of entities in a system 


model. These events can be easily Gonverted to the events of 
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an event-driven simulator. Another valuable feature of an 
event-driven simulator is the capability for pseudo-coincident 
simulation activity. This is required for network simulation 
where several computer system nodes may be operating in 
parallel. | 

An event-driven simulator can easily be implemented with 
a clearly defined modular structure. This structure is 
illustrated in Figure 5.4a. The modules can be divided into 
two groups: those that maintain the evene? list. 
(initialization, event selection, event scheduling and event 
removal) and those that do event activities (the event 
procedures). The event list maintenance modules are 
independent of the type of system being simulated. The event 
procedures on the other hand are specific to a particular type 
of system (eg. HPC network systems). By insuring the isolation 
of system dependent features, the Simulator can be 
straightforwardly adapted whenever the system is modified. 

The structure of the event list is the key to 
pseudo-coincident simulation activity. It is the place where 
the simultaneous activities Shearing throughout a simulated 
network are merged into a serial stream of events. The MNSS 
event list structure is shown in Figure 5.4b. Each event list 
item carries enough information to identify the event type and 
where in the network system the event is to occur. The event 
list is linked together with the next event to occur placed at 


the head of the list. The foremost event is pointed to by a 
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“next event pointer’. Event list items that identify 
coincident events are arranged with the first-scheduled events 
being foremost in the event list (ie. first to happen by way 
of the event procedures). This procedure yields an adequate 
ordering of events for MNSS studies of HPC network systems 
(where events are frequent and limited in scope). 

Appendix A discusses the MNSS/3000 implementation which 
incorporates the ideas presented in this chapter. There is 
also additional generally applicable friformation on the 
implementation of MNSS. The MNSS/3000 implementation is used 
to provide the simulation results’ that are used din the 


following chapters on verification and experimentation. 
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CHAPTER 6 
VERIFICATION OF THE MNSS IMPLEMENTATION 


In order for simulation results to be useful they must 
represent a verifiable picture of the behavior of the system 
being simulated. Clearly it is not realistic to compare the 
results of every simulation experiment to measurements of the 
eorreaponding real system. This would defeat the purpose of 
the simulation system to be more adaptable for study then a 
real system. In addition a simulation system may be used to 
anticipate the behavior of a syster yet to be developed. In 
this case no Measurements are possible, hence other 
verification techniques must be applied. 

Effective verification procedures utilize a limited 
amount of information about the predicted or measured behavior 
of a system in order to prove that simulation results (in 
general) are valid. Among the Ooealves that are often used 
are the identification and verification of simulation model 
parameters and the functional verification of simulation 
system operations. 

Simulation model parameters represent real system 
constants that can be isolated from complex system 
interrelationships. They may be anything ranging from the 
speed of a communications link to the time it takes to do a 
particular task (eg. start an execution of a process once it’s 


in main memory). The verification (where it is necessary) of 
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- parameters used in the simulation model can be done through 
_Measurements or the prediction of system behavior. 

: The functional verification of the simulator requires 
_ matching behavior in the simulated system to behavioral data 
produced by simulation. This can be done by first insuring 


that basic behavior patterns in the simulated system are 


ta e 


represented in the simulation model; if action “xX always 


e id 


“follows action Y in the simulated system then the same 
sequence must be represented in the model. Then the behavior 
pattern results produced by the simulator wee verified in view 
of corresponding results generated by the simulated system. 

Functional verification can be done at two levels. 
First, fundamental behavior pattern results can be predicted: 
for a particular system and then confirmed for the simulation 
system. This verification can be done to the extent that 
fundamental behavior patterns occur in a system and produce 
predictable results. Another level of functional verification 
can be done through the use of system measurements. 
Unpredictable behavior pattern results can be identified and 
compared to simulation experiment results. | This form of 
functional verification must be done with discrimination 
because there are potentially an infinite number of behavior 
patterns. | 

The following two sections of this chapter will describe 


the procedure by which MNSS is verified in light of HPC system 


behavior. Because HPC network systems are in a development 
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stage, no measured results are available, but measurement 
experiments are specified for later use. For now verification 
of critical model parameters and functional behavior are 


dependent on predicted system behavior. 
6.1 Simulation Model Parameters 


For the HPC system model three groups of parameters have 


been identified: 


1. system process completion times, 
2. process switching overhead, 


3. Operating system constants. 


The values of the parameters in these three groups are either 
constants or determined by a known function (of environmental 
factors). | | 

There are two types of communications system processes 
represented in the HPC eyaten model. In an HPC system these 
system processes have associated modules of code that do link 
control and message handling. The “generation of a system 
process results in a reasonably predictable execution path 
through the code modules. By taking into account the code 
execution times of an “HPC system and the nature of the 
execution paths, a prediction can be made for the system task 


completion times. Based on this procedure the following 
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completion times have been derived: 200ms for the message 
generation system process, and 100ms for the message routing 
system process. 

In order to get more accurate values for system process 
completion times, measurements could be done on an HPC network 
‘system when HPC systems and measurement tools become 
available. The measurements would require a message source, a 
software monitor, and at least a three node HPC network (to 
have message routing). The software monitor would take timings 
at the end points of the execution paths through system 
process code modules. These measurements (repeated many 
times) would ‘yield representative values for system process 
completion times. | 

Process switching overhead can, in general, be 


represented by the following expression: 
Tov = Cl + C2*Min + C3*Mout 


In this expression, process switching overhead (Tov in units 
of time) is a sunction of the memory requirements of the 
processes involved in the switch (Min and Mout in units of 
space), and the system code that must be executed to do memory 
management (C2 and C3 in units of time/space) and CPU control 
transfer (Cl in units of time). System code execution times 
can be predicted based on an examination of the execution 


paths associated with process switching. For the first 
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implementaion of the HPC system model (ie. the implementaion 
referred to in this thesis) a standard memory requirement is 
assumed for all user processes so the expression for process 
switching overhead reduces to a constant (100ms is the derived 
value of the constant). This simplification of the overhead 
function is necessary because of the difficulty to predict its 
value in general. 

Measurements on an RPC system will be needed to 
accurately define the parameters in the overhead function. 
These measurements will require an HPC system, a software 
monitor, and a controlled user process load on the system. 
The measurement procedure would involve generating two user 
processes with a known memory requirement and then allowing a 
specified number of process switches to occur. By measur ing 
the overhead associated with a variety of Srébens memory 
requirements enough values could be generated to identify the 
overhead function: 

Operating system constants are set to a specific value 
when the operating system is implemented. Therefore no 
measurements of HPC system operations are needed. Some 
operating system constants are fixed for all HPC system 
implementations. For example, the minimum CPU control time for 
a user process before it can be preempted by another user 
process (ie. the ‘user slice”) is set to 500ms. On the other 
hand some constants are particular to an installation and must 


be redefined for each different HPC network. The window size 
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for a half duplex communications line is an example. 
6.2 MNSS Functional Verification 


There are a number of fundamental behavior patterns that 


are inherent in an HPC network system. These include: 


1. continuous CPU bound execution with a standalone user 
process (ie. there is no process switching overhead), 

2. long wait initiation by a process, removing the 
process from contention for CPU control, 

3. process awitching and the consequent overhead: 

4. system processing resulting from network message flow, 

5. the startup of a remote user process after receipt of 
a remote request, 

6. the completion of a remote user process resulting in 
the generation of a remote response message, 

7. local user process restart oa “receipt of a remote 
response, | | 

8. store-and-forward message traffic in networks that are 


not fully connected. 


Each of these behavior patterns should be duplicated in the 
behavior of the simulated HPC network system. To show that 
this is indeed true specialized simulation experiments can be 


defined that focus on a particular pattern of behavior. The 
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exper iment specifications must describe a system whose 
behavior can be predicted. If the simulation experiment 
results are as predicted then the fundamental behavior pattern 
that produced the results is verified. The experiment 
analysis that follows is part of the process to kgnccienaliy 
verify MNSS. | 

In order to verify the functioning of process switching, 
remote request operations and store-and- forward waieaiene. a 
standard simulation experiment has been specified. For each 
MNSS function to be verified, appropriate user process 
definitions are added to this specif ication. Appendix B gives 
a summary of the standard experiment in the MNSS descriptive 


notation. 
6.2.1 Process Switching 


The | simulation experiment used to verify process 
switching has two CPU bound (no long waits) user processes 
competing for CPU control. Neither of the user sroceuuee ever 
makes a remote request so all devivity in the simulated system 
takes place at one HPC network node, The activity in the_ 
system can be described by the following state-time sequence: 

time > * 100ms * 500ms * 100ms * 500ms * actions repeated 
Process 1 * wait * CPU * wait * wait * as ae 
Process 2 * wait * wait * wait * CPU * a 
The time intervals that occur when both processes are in the 


CPU-wait state represent the times when there is process 
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switching overhead. Given this simple behavior pattern and 
knowledge of the length of time tie sinulation spans, it is 
easy to predict CPU utilization and CPU-wait statistics. For 
example, durning a one minute: period the process switching 
overhead is derived by dividing 60000ms by 600ms (the period 
that includes one process switch) and multiplying the result 
by 100ms (or the time calculated to do aie precees switch for 
the two processes’ involved). The overhead in this case is 
equal to 10000ms. | 

The predictions are confirmed by the statistics generated 
by an MNSS experiment “(given in Appendix C). Thus the 
fundamental process. “gereching mechanism works -for MNSS 


experiments. 
6.2.2 Remote Request Functioning 


There are eight distinct phases to the cycle of behavior 


a user process exhibits. in making remote requests: 


l. a user process does local activity (processing and 
long waits), 

2. the process initiates a remote request and an 
appropriate message is generated, 

3. the remote request message is _ transmitted to its 
destination, 


4. a remotelly initiated process is generated, 
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5. the remotelly initiated process runs to completion, 

6. the remotelly initiated process completes and a remote 
response message is generated: | 

7. the remote response message is transmitted to the node 
of the remote request originator, | 

8. the remote request originator is restarted for local 


activities. 


Remote request behavior can be isolated and verified by 
an experiment with a CPU bound process making remote requests 
to a neighboring node (ie. no store—-ané- forward) . Given a 
remote request interval of 500ms, message sizes of 100 bytes, 
a link rate of 8000 baud, and a remote process completion time 
of 500ms, the following cycle can be predicted: 


Time(ms) > 600 * 200 * 100 * 100 * 600 * 200 * 100 * 100 
Phase pe? a” ee ae a, a «ne 


Using this cycle, system statistics can be derived for a one 
process, two nods system. For example, durning a one minute 
period this cycle (2000ms) would be repeated thirty times. 
Therefore local user CPU processing and overhead (phase 1) 
should be 30x600=1800mrs and remote CPU processing and overhead 
(phase 5) should also be 1800ms. These and other predicted 
‘phase’ statistics verify the simulation experiment results 


(Appendix C). 


6.2.3 Store-and-forward Functioning 


The experiment used in Section 6.2.2 can be modified to 
show store-and-forward behavior. Instead of having remote 
requests sent to a node connected directly to the originator’s 
node, they can be sent to their destination through an 
intermediary. Phases 3 and 7 of the behavior cycle given in 
Section 6.2.2 have to be ex panded to account for 


store-and-forwarding: 


| ee 

3. transmit request message to intermediary node, 
3-1. process message and rout it to the appropriate link, 
3-2. transmit message to request destination node, 

4-6 ... 

7. transmit response message to intermediary node, 
7-1. process message and rout it to the appropriate link, 


7-2. transmit message to response destination. 


The remote request cycle time is lengthened to 2400ms 
(using the Section 6.2.2 system parameters) with the changes 
to phases 3 and 7 shown below: 


Time>..100 * 100 * 100 * 100 *..* 100 * 100 * 100 * 100 
Phase>.. 3 * 3-1 * 3-2 * 4 *..% 7 ¥* Jol * 7-2 * 8 


The modified remote request cycle can be used to derive 
statistics for a system that has. store-and-forward behavior. 


Durning a one minute period the cycle would be repeated 25 
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times. This means that the store-and-forward system 
processing overhead (3-1 and 7-1) is equal to 2x100x25=5000ms. 
This verifies the statistic for system processing in the 
store-~and-forward node generated by simulation (summarized in 
Appendix C). _ | | 
Functional verification experiments must be used 
initially to verify the simulation eceten and then whenever a 
change is made to. the structure of the simulation model. 
Therefore an extensive set of éxperimenta is witel to insure 


correct MNSS operation. 
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CHAPTER 7 
MNSS EXPERIMENTS 


Properly iaovencnted: MNSS provides a Honvenient tool for 
interesting simulation analysis. The goal of this chapter is 
to demonstrate the capabilities of MNSS 440 particular 
MNSS/3000 - nee Append ix A). An experimental process is 
presented that meshes with the mechanisms of MNSS. This 
process, which is essentially the wideiy aclained ‘scientific 
method’, gudaee. the MNSS tibet | in doing effective 
experimentation. The latter section in this chapter presents 
an outline for two simulation studies that are interesting in 
light of commercial computer network applications. Sample 
results from the two Stuer speCdncee using MNSS/3000, are 


given wien brief analyets: 
7.1 The MNSS Experimental Process 


There are virtually an unlimited number of simulation 
experiments that can be done using a fully implemented MNSS, 
From this profusion of experiments the MNSS user must select 
those that most directly serve “his/her purpose. this selection 
procedure can be fotmalized: into the MNSS Experimental Process 
which, faithfully used, ef fectively focuses experimentation in 
an MNSS simulation study. Ee | 


The MNSS Experimental Process is summed up by the 
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following list of steps: 


1. Form a hypothesis dealing with the behavior of HPC 
network systems; 

2s peaign ‘i set of experiments that conceivably will 
demonstrate the veracity of the hypothesis; 

3. Build a number of key experiments that can be used to 
generate, by way of MNSS experiment modification 
facilities, the entire set of interesting experiments; 
and | 

4. Run the experiments, analyze the results and determine 


the veracity of the hypothesis. 


These steps can be repeated several times with continuing 
adjustment of the hypothesis to account for experimentally 
produced information: 

The MNSS experimental process can be used to structure a 
broad range of simulation studies. For example, to explore 
basic networking structures. a hypothesis might be: “Half 
duplex communications lines require significantly more message 
buffering for high message traffic than do full duplex lines.” 
This hypothesis can be refined eircdgh expecdnantacion to 
state exactly the relationship between line protocol, line 
speed, message traffic, and message buffering. Another type of 
study involves network per formance optimization for a 


particular application. In this case the hypothesis might be: 
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"For distributed data processing application “N’ the most 
cost-ef fective network configuration is a four node star, with 
9600 baud full duplex communications lines, etc." Ref inement 


of this hypothesis leads to an "application cuatcesec? network. 
7.2 Two Simulation Studies 


The following simulation studies were done using 
MNSS/3000. . The goal of the experimentation was to demonstrate 
the capabilities of MNSS. ” Due to the ‘Limited nature of the 
verification of MNSS/3000 (Chapter 6) the reaulte from the 
studies should be regarded with caution, ‘The results reveal © 
general forms ee HPC network behavior and are not: predictive 
of the exact behavior of any" particular network. When 
MNS S/3000 is verified and tuned for a particular HPC system 
(eg. HP3000) the simulation results generated will be more 


accurate and specific in showing forms of network pelayior. 

7.2.1 Incremental Network Expansion 

Hypothesis: The incremental processing increase achieved by 
adding a node to a network is sensitive to the 


network’s line and node characteristics. 


This was an awesome hypothesis to confront head-on, so 


some simplifying considerations were in order. First, only 


90 


fully connected networks were eeae nea: each node in the 
networks of interest had to be Linked - every ether node by a 
communications line. Second, the communictions lines in a 
particular network all had the same specifications (ie. duplex 
type and line speed). Third, identical standard user loads 
(based on a standard user process aceciticecion) were used at 
each node. Forth, the pereveters varied in Gersning the user 
load ee a network node were limited to the number of Permanent 
user "processes (formally described: in Chapter 4, pp. 54) and 
the remote request. level per user process. The permanent user 
BECcenees were the agents: of transaction data prdceasing; they 
were Honk tHueaw ly active (which assumes. an infinite supply of 
transactions to be processed} . All « were  @ef ined with identical 
lotal processing characteristics (ie. ‘computation, short wait 
and - long wait réequiremente) . In addition the dest ination of 
‘remote requests originated by a . permanent user process had 
| equal probabilities of being any non-local node in the network 
(ie. a central database application load was not considered). 

Based on the hypothesis and the associated limitations a 
set of experiments was designed with a variety of network 
configurations, auphex TYPES» line speeds, network processing 
levels, and remote request levels. Bach experiment in the set 
was specified by drawing parameter calues from the following 


list of alternatives: 
l. network configuration - two, three and four node fully 
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connected networks; 

2. duplex type - half and ‘full duplex; — 

36 line speeds - 1200, 9600 and 19200 baud; . Ane 

4. network processing level ~ 12, 18 and 24 permanent user 
processes in the networks oo 


5. remote request level - low, medium and high. 


There are 108 potential experiments thet can be built with 
different combinations of ehese: alternatives. 

The low, medium and high pacing for the remote request 
level are based on the expected Process ind. that occurs locally 


before a remote request is initiated (ie. the local processing 


time of a user process from the completion of one remote 
_ request to the initiation of | anothers ENG: bine between 


remote request’ function is (1-P) T/P, where P is the 


PEObaDsTTEY of a remote Zequest and T. is the transaction time. 
The transaction — time is the amount. of time (in milliseconds) 
required to do some basic amount of processing (ie. a 
transaction). Each ‘transaction time’ of processing done by a 
user ‘process is directed either locally or remotely (ie. a 
remote request). Single accesses to a “database can be 
classified as peaneacetoner taking a. specific _amount of 
processing to donpieke: The rewate request probability is 
defined for each user process, and is the probability of a 
transaction neeaing remote processing (eg. ene ae five of the 


transactions originated by user process A requires remote 
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processing; this implies a remote request probability of ie 

Figure 7.1 shows the table Be Gace process specification 
values for the time between remote requests. For the 
experiments at hand, a transaction time of 250ms was always 


used. 


7.2.1.1 Experiment Results One 

Some interesting experiment results are presented in 
graphical form in Figure 7.2 (a, b and c). The simulation 
experiments that generated these ‘results were run fora 
simulated time of 600 seconds. Up to 20,000 events were 


generated with the running of an experiment (four nodes, high 


' remote request. level, 24 user processes) . Using a timeshared 


HP3000 this took 20 seconds of real time. For the results 
given here all networks were specified to have 9600 baud full 
duplex lines: ‘ana had loads of 12 or 24 user processes. 

‘The processing performance of the networks studied can be | 
measured in terms of transactions per second (ie. the network 
transaction processing throughput generated by the specified 
permanent user processes). Figure 7.2a shows the relationships 


(simulation derived) between the number of network nodes, the 


remote request rate and the transaction processing throughput. 


Trend lines have been drawn in to highlight these 
relationships. These lines illustrate in a very rough way the 


functions associated with various groups of data points. 
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remote request level 


LOW MEDIUM HIGH 


25 
5 
transaction 
times 
(seconds) 
2 


all values 
in seconds 


Figure 7.1 Remote Request Level Table 
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remote request level 


~ low 
2. “ low 
medium 
transaction high 
processing 
throughput. 
(tran/sec) 
b. 
100 
80 
network 60 
utilization h medium 
% 0 
20 IS ee ee a et ON 
i a eyed a 
2 3 us 
# nodes 
c. 
network 
idleness 
(% idle) 
# nodes 
weceoe 12 user processes ------ 24 user processes Lines: 9600 baud, full 


Figure 7.2 Incremental Network Expansion Experiment Results duplex 
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Figure 7.2b puts the transaction processing throughput 
values shown in Figure 7.2a into perspective. Network 
utilization is measured as a percentage of the theoretical 
maximum transaction processing throughput (where there is no 
system overhead due to conmunioations Ot pencaus switching). 
The maximum value for 4 D50ke evaneaction time is 1/.250 = 4 
~ transactions per | second for one _ node Or 4xN 
transactions/second for an N node network. 

‘The results given in Figure 7.2a clearly show that 
network | transaction processing throughput increases with 
incremental network expansion, but that this increase. 18 not 
proportional to the number OF nbaes added to the network. As a 
network increases in. sive, network transection prosensiad 
efficiency (per node) can decrease due to communications 
overhead or lack of work to do (assuming the number of user 
processes specified ie. Gohatant) Pidure: Te2¢ shows that for 
the network situations examined here the falling utilization 
(with incremental growth) is aide kouscae degree to idleness; 
not enough processing work was available to keep all nodes 
busy all the time. 7 

A partial solution to the idleness problem is also shown 
in the Figure 7.2 results. An increase in the number of user 
traneaction processes led to decreased idleness (especially in 
3 and 4 node networks) and increased network utilization. This 
illustrates the sensitivity of network utilization to the 


character of the network user load. 
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Another potential solution to the iaiefees problem was 
examined and found to be ineffective. This approach entailed 
increasing the line speeds in the network from 9600 to 19200 
baud, wnilevmsintaining a user load with 12 processes. If the 
idleness was due to Communications delays then this would 
decrease idleness and increase network utilization. No 
significant increase in network utilization was shown when 
experiments were done; the idleness was not due. to. 
commun ications delays, but. rather to delays associated with 


system processing. 
7.2.2 Four Node Network Processing Characteristics. 


Hypothesis: The total processing capabibility of a four node 
| network . ‘is sensitive to node connection 


characteristics and the applied user load. 


This is vanothes sweeping hypothesis chat. haa’. tobe 
diluted in scope in order to design initial experiments. The 
simplifying considerations were the same as- those of the 
“incremental network expansion hypothesis’ with two 
exceptions. First, star and node -eonf igurations were used in 
addition to a fully connected configuration, Second, the 
ienote request agetinacton distributions in some experiments 
were indicative of a centralized network database application. 


For these experiments one node was designated the database 
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processor (usually the central node in a star network and an 
arbitrary node in ring and fu1iy-connected networks), and most 
of the remote requests were directed to this node. In addition 
the user processes in the database node were primarily 
processor bound (ie. deine database maintenance) and 
infrequently initiated remote requests. 

The set of experiments based on the simplified “four node 
network processing” hypothesis was designed with various 
network configurations, line speeds, network processing 
levels, remote request levels and remote request destination 
distribution types (indicative of centralized aiid distributed 
network databases). The experiments in this set were specified 


using the following list of alternatives: 


1. network cone guration - ese ring and fully connected 
four node networks; | 

2. duplex type - full duplex; 

3. line speeda - 1200, 9600 and 19200 baud; 

4. network processing level - 16 and 24 user processes in 
the ee 

5. remote request level - low, medium and high; 

6. remote request distribution - centralized and 


uncentralized. 


Using these alternatives 108 different experiments can be 


built. The remote request level parameter is specified using 
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the table shown in Figure 7.1, with a transaction time 


of 250ms. 
7.2.2.1 Experiment Results Two 


Selected results from “four node network’ experiments are 
shown in Figure 7.3 (a and b). The results given in this 
section are dravn primarily from distributed database network 
experiments. The networks were specified with 9600 baud full 
duplex lines and had loads of 16 user processes. Experiments 
using centralized network database specifications yielded 
inconclusive results and need to be supplemented by further 
experimentation. As with the results described in Section 
7.2.1.1 the results presented here were generated by 
experiments that ran for a simulated time of 600 seconds. In 
addition, the ‘transaction processing throughput’ graph in 
Figure 7.3a measures the same quantity, in the same units, as 
the graph in Figure 7. 2a. 

The results shown in Figure 7.3 graphically illustrate 
the effect of communications overhead on transaction 
processing throughput. The effect is most noticeable when 
there is a high remote request level. With the star 
configuration there is a higher communications overhead than 
with a ring configuration, which in turn is higher than the 
overhead in a fully connected configuration (since in a ring 


the processor also has to perform message routing for messages 
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a. remote 


= request 
12 level 
low 
po G ia 
10 es ee ee medium 
transaction 8 
processing 
throughput _ 7 
(trans/sec) 6 ae 7 amie 
. 4 
2 
STAR RING =———sSFULL 
configuration 
b. 
. ‘STAR 
5 (central node) 
h 
CPU wait 3 
(seconds) 
FULL 
2 
1 
STAR 
(point nodes) 
low med lum ~ high 
remote request level 
‘Lines: 9600 baud full duplex Loads: 16 user processes 


Figure 7.3 Four Node Network Experiment Results 
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destined to other nodes). This is one cause for the 
relationships of throughput values in Figure 7.3a 
(particularly for the high remote request Level relationship). 
There is a processing (system and eer) bottleneck in the 
center of a star configuration caused ‘by communications 
overhead. Figure 7.3b shows hew long on the average a user 
process waits in the CPU queue before it gains control for a 
“processing quantum. The wait in the center of a star network 
is significantly higher than the wait at the points of a star, 
or in any node in a fully configured network. At the center of 
a star there is a great deal of message traffic that results 
in communications processing and chronic preemption of user 
processing. . , 

The communications processing disadvantages of the star 
conf iguration wane into insignificance when the remote request 
‘level is reduced. As is shown in Vieure-7.3a; the star 
configuration is virtually equivalent in terms of transaction 
socessiaa throughput for a low resiote ‘aquest level. The 
-disappearence of the bottleneck with a decreasing level of 
remote cauues ta is confirmed: by the results given in 
Figure 7.3b. 

The. results described in this ‘gaction reflect only a 
small portion of -the ‘information generated by “four node 
network’ experiments. For example, the results of experiments 
' done with central database specifications indicate that a star 


network can sometimes operate as effectively as a fully 
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connected network even with high remote request levels (this 
is because most requests ‘are directed to the central node). 
Further experimentation and -analysis is needed to provide a 
Clearer overall picture of network performance with 
centralized database applications. In any case the results 
documented here demonstrate what can be done when MNSS is used 
within the context of a clearly defined network analysis 


study. 
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CHAPTER 8 — 
CONCLUBIONS - 


The development of MNSS- led to a nyeee® of insights and 
_ accomplishments that made the effort wor thwhile. The initial 
careful examination of esesaitee network systens identified a 
need for analysis tools that could ber Need to explore the 
complex behavior associated with these systems. MNSS eae then 
developed to help meet this need. 

MNSS is a multifaceted system that includes a modelling 
discipline, the building blocks for a set of minicomputer 
network models, and a simulator. The MNSS modelling discipline 
is simple to use but very powerful as a method of providing 
abstract representations of system structures. Using the MNSS 
modelling discipline building blocks were developed for set of 
minicomputer. network models. The principle building block is 
-an HPC system model. The modelled minicomputer networks are © 
collections of HPC systems. The rising popularity of network 
nysteue (including HPC network systems) makes an understanding 
of thei benavior a valuable commodity. The MNSS simulator is 
the principle agent in providing HPC network system 
performance data. The flexible, easy to use characteristics of 
the simulator are what make MNSS a worthwhile system analysis 


tool. 
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8.1 Limitations Of MNSS 


There are a number of limitations to MNSS in its current 
state of development. These limitations encompass both easily 
correctable functional shortcomings and also more fundamental 
design problems. 

There is no formal procedure yet def ined that can be used 
to initialize a network model in such ‘a way as to avoid 
noticable transitory effects when the model is used for 
simulation. Currently with certain network situations the 
simulation statistics generated for maximum queue lengths and 
maximum queue waits reflect the initial setup of the model and 
not the overall system behavior that is of interest (eg. 
starting with | all permanent user processes in the CPU-wait 
queue may produce a system atate that does not occur under 
realistic load conditions). 

There are two potential solultions to this problem. One 
solution is to Achawe in’ the permanent user load while doing 
simulation with a system model. This would entail inserting 
permanent user processes into the CPU-wait state over a period 
of time instead of all at once when simulation begins. For 
this to be successful the phase in time would have to be short 
compared to the total simulation time. A second solution is to 
begin gathering initialization sensitive statistics only after 
all startup effects have disappeared. The simulation time 


after the delay in statistics gathering should be sufficiently 
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long to insure that the final simulation statistics are valid. 
The usefulness of MNSS results is now limited by the 
incomplete process of HPC system model verification. 
Completion of the verification sronedure presented in 
Chapter 6 will result in proving the ‘model correct for a 
particular HPC system; this is required before MNSS results 
can be used vith complete confidence. In addition since the 
HPC system model can be used to represent a class of systems 
(ie. HP produces a series of Edeeuters with related 
scchitecturess; there must be an “adiuscaont and verit ication 
of the model for each type of HPC system to be analyzed in a 
simulation study. | | 
There are a number of limitations in the current 


implementation of the HPC model description. These include: 


l. the model description lacks a specification for computer 
system primary memory sizes that can be used in 
calculating CPU-control process switching euecneua, 

2. there is no convenient ae. to weds! networks of computers 
with “different processing speeds (ie. all computers ina 
network must either have the same processing speed or the 
user load specification must be adjusted to sccount for 
adifferences), and | 

3. the CPU-control process switching: overhead does not 

| directly take: into account secondary memory to primary 


memory transfer rates (there should be a parameter to 
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specify this rate for each system). 


_ These limitations can easily be overcome and will be 


| eliminated with the next stage of MNSS development. 
8.2 Extensions To MNSS And Further Study 


With additional Study and development a number of 
extensions to MNSS could be made to ineranse ite capabilities. 
The extensions discussed in this section do not necessarily 
7 require major modifications to MNSS as it is currently 
- implemented. | | 

With some changes to the user load specification part of 

the MNSS simulator, trace date compiled by monitor ing a 
particular application could be used directly to specify the 
user load for a simulation experinent. This eeecure would be 
very useful in doing a simulation study aimed at optimally 
conf igur ing a BEEMOEK for an existing. application (that is the 
application is cunbitng on some available system). | 
Selective submodel simulation is not now provided by 
MNSS. By extending MNSS ‘s include this capability, users 
would have more control in doing ‘simulation studies. MNSS 
could be used to simulate network ueneage traffic with a full 
HPC network model or with only the communications submodels 
(messages would be generated at message sources’ using 


frequency distributions). The . principle difficulty in 
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implementing selective submodel simulation is how to. 


characterize the universe as seen by the selected ubmose?: 


This aspect of the universe must be reduced from an active . 


simulation model to an analytic furiction or trace data (the 
most likely solution). This must be done dynamically when a 
user selects a submodel for simslation, 

Models | are now described interactively using predef ined 
building blocks made from the basic mocek discipline 


structures (eg. groups, entities, ete: )e ‘These building blocks 


(eg. processing submodels, half/full duplex communications 


‘structures, etc.) are integral parts of an implemented MNSS; 
there is no possibility of ‘simulating different types of 
systems (eg. computer networks and aupermarrsts) Additional 
‘ecday: into the conversion of MNSS into a generally applicable 
simulation system (GASS?) is a worthwhile endevor. In 
‘particular, capabilities to interactively describe simulation 
models using basic MNSS modelling discipline structures can be 
devcioned: “af the capabilities of MNSS are extended in this 
way then a _ new range of simulation possibilities arise. The 
“scope of these possibilities can be determined to some extent 
by determining the relationship of the MNSS mode tt +09 
discipline to. other discrete modelling disciplines. The MNSS 
modelling discipline is very general and may include many (or 
all) of the modelling capabilities of chess other disciplines. 
The potential for enhancement and use of MNSS appears to be 


virtually boundless. 
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APPENDIX A 
THE MNSS/3000 IMPLEMENTATION 


MNSS has been implemented in SPL on an HP3000 computer 
system. SPL is an ALGOL-like language, which has constructs 
applicable to structured programming (eg. while ... do ..., if 
oes then ... else ..., etc.). These constructs are used 
extensively in the MNSS program for top-down structuring and 
modularization of function. The P3000 on which MNSS was 
implemented supports software such as SPL and mathematical 
library functions (eg. natural log, randbe number generator a 
and hardware such as disks, CRT’s and lineprinters. This 
support was necessary in order to take advantage of the 
interactive and simulation capabilities of MNSS. 

This implementation of MNSS (MNSS/3000) has’ several 
characteristic features that reflect its versitility as.a 
system analysis tool. These features include the selection of 
MNSS function options from menus of alternatives, intelligent 
dialog interaction, disk storage of simulation experiments, 
and hard-copy MNSS informacion display. These features are 
needed not only for MNSS/3000, but also for any effective 
implementation of MNSS. 

A MNSS/3000 function menu is prefaced by the statement 
“SELECT OPTION", and is composed of a list of alternatives 
with associated numbers (0-n). The user is prompted for a 


reply by a ">", and selects an alternative by entering a 
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number. If the user enters a number that does not correspond 
to an alternative in the menu, then “INVALID RESPONSE" is 
Output and the user is again prompted. A valid response will 
result in the activation of the corresponding alternative. 
Another form of interaction used in the. MNSS/3000-user 
interface is the responsive dialog. MNSS/3000 outputs a 


question, prompts the user with ">" and then waits for a 


. response. The response is evaluated by MNSS/3000 and a proper 


followup is output. The goal of the responsive dialog is to 
channel information to and from MNSS/3000 without irrelevent 
(dumb) questioning. 

The nature of the display and storage of simulation 
experiment information are very important in determining the 
usefulness of any simulation system. MNSS/3000 provides the 
capabilities to store and retrieve simulation experiment 
specifications represented as disk files. A user can avoid the 
trouble of respecifying _an experiment each time it is to be 


run. In addition, there may be standard experiment kernels 


that can be built upon to produce desired experiments. These 


kernels can be kept on disk and retrieved whenever necessary. 
Lineprinter output of MNSS/3000 information is provided at the 
user’s direction to supplement the normal form of interactive 
output. In many cases this is a desireable alternative to 
information display at interactive station (eg. CRT’s produce 
no printouts and teletypes often take ages to produce a low 


quality printout). 


A MNSS/3000 experiment is constructed in line with the 
principles discussed in Section 5.2 (‘Representation Of A 
Simulation Experiment’). It can be used by the simulator to 
generate behavioral information, saved for future use, oz 


modified to yield related experiment specifications. 
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APPENDIX B 


VERIFICATION EXPERIMENT 


ReARRKRRKRKEKERRKRE 


* * 
» NETWORK DESCRIPTION , 
kek RRR RHR KA KK 


NODE CNT=3 
LINK CNT=4 
LINK CONNECT DUPLEX RATE WSIZE DELAY 
0 0->1 FULL 8000 
1 150 FULL 8000 
2 1->2 - FULL 8000 
3 2->1 FULL 8000 
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* : * 
» MESSAGE ROUTING , 
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Note: there are no uniform, special, exponential or destination 
distributions specified for the verification experiment 
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ARK RAR A 
* 

y CONSTANT DIS 
kkReEK KK KK 


ENTRY REFER. VALUE 
0 1007. 20000000 . 
1 1002 100 
2 1001 500 


+e + 


eae KKK KRKE KR 
* * 
 CHARACTERIZATIONS , 
kKaeARkKK RRR RE KK 


"ENTRY REFER CTIME CPUT LONG MEMC MSGL 
0 1002 CO co. 0cOC«*O Cl 
1 1002 c2 cO co 1 Cl 


kKaxakREA 


*% 
» PROCESSES 
kRakeaR KAR KE 


6.2.1 Process Sharing 


+e e & 


ENTRY NODE TYPE CHAR RDIS RDES RCHAR . 
0 0 0. 0 COo 2 - obs 
1 0 0 ) co 2 1 


RekRR EE 


* 
» PROCESSES 
RRR RK 


6.2.2 Remote Request Functioning 


+e e 


ENTRY NODE TYPE CHAR RDIS RDES RCHAR 


0 0 0 0 c2 1 1 
Kwek k OR 
* * 
» PROCESSES , 6.2.3 Store~and-Forward Functioning 
kKkRkEH KER , 


ENTRY NODE TYPE CHAR RDIS RDES RCHAR 
0 0 0 0 C2 2 1 
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APPENDIX C 


EXPERIMENT RESULTS 


RRR RK KER KE KR KEK KKK KKK 


6.2.1 PROCESS SWITCHING OVERHEAD 
RRR KKK KKK KKK KK KKK 


+ ee 
+ +e FF 


ee FF 
* 


*k * & & * UTILIZATION TIMES * * * * * 
NODE SYSTEM LUSER HUSER OVERHD IDLE 


0 0 50000 0 10000 0 
1 0 0 0 0 60000 
2 0 0 0 0 60000 
kK KKK KK 
k x 
y CPU WAIT , 
kK KK KKK 


NODE MAX-SIZE MAX-WALL ENTRIES TIME*LEN 


0 2 700 100 70000 
1 0 0 0 0 
2 0 0 0 0 
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TOT-WAIT 
69400 

0 

0 


+ + 


LINK CONNECT IRMIT IDLE 


0 0-1 0 60000 
1 1-0 0 60000 
2 1>2 0 60000 
3 2>1 0 60000 


kRRKKE KKK K 


* 
LINK WAIT , 
kK Kk KKK 


* 


LINK CONNECT  MAX-SIZE MAX-WAIT ENTRIES TIME*LEN TOT-WAIT 


0 0>1 0 0 0 0 0 
1 1>0 0 0 0 0 0 
2 1>2 0 0 0 0 0 
3 2>1 0 0 0 0 0 
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* 
6.2.2 REMOTE REQUEST FUNCTIONING , 


RKaRKRE KKK KKK KEK KR RK KK KK 


+ ee 


+ +e t + 
+ 


* * * & * UTILIZATION TIMES * * * * * 


NODE SYSTEM LUSER RUSER OVERHD IDLE REMPR 


0 9000 15000 0 3000 33000 15000 
i 9000 0 15000 3000 33000 0 
2 0 0 0 0 60000 0 
kKkKREK KKK 
* k 
, CPU WAIT , 
RKRKREKR KK EH 


NODE MAX-SIZE MAX-WAIT ENTRIES TIME*LED TOT-WAIT 


0 a 100 30 3000 3000 
1 1 100 30 3000 3000 
2 0 0 0 0 0 
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wk KKK EK 
* * 
, LINK , 
kk KKK 
LINK CONNECT 

0 0->1 

1 1->0 

2 1->2 

3 2->1 
KKK KKK 
* x 
y LINK WAIT , 
KKK KK KK 
LINK CONNECT 

0 0->1 

1 1->0 

2 1->2 

3 2->1 


TRMIT IDLE 
3000 . 57000 
3000 57000 
0 60000 
0 60000 
MAX-SIZE §MAX-WAIT 
1 0 
1 0 
0 0 
0 0 
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ENTRIES 
30 0 
30 0 
0 0 
0 0 


ITEM*LEN | 


TOT-WAIT 
0 


0 
0 
0 


ee + F 


+e FF 
+e e 


6.2.3 STORE-AND-FORWARD FUNCTIONING 


RaAReRKEKK KK KER KKK KK KKK KKK 


KAKA KKK KK KR KKK KK KKK KKK 


* 


*& * & & & UTILIZATION TIMES * * * * * 


NODE SYSTEM 
0 7500 
1 5000 
2 7500 

RAK KK KKK 

* * 

~ CPU WAIT | 

kRKK KK KK 


NODE MAX-SIZE 
0 1 
1 0 
2 1 


LUSER 
12500 
0 
0 


RUSER OVERHD IDLE 


0 2500 37500 
0 0 55000 
12500 2500 37500 


ENTRIES TIMEXLEN 
25 2500 
0 0 
25 2500 
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REMPR 
12500 
0 
0 


TOT-WAIT 
2500 
0 
2500 


pe a 


LINK CONNECT 


0 0->1 
1 1->0 
2 1->2 
3 2->1 


kKkRKK KK 


LINK WAIT 
KK Kk KK 


ee bE 
ee OF 


LINK CONNECT 


0 0->1 
1 1->0 
2 1->2 
3 2->1 


IRMIT 
2500 
2500 
2500 
2500 


MAX-~-SIZE 


1 


1 
1 
1 


IDLE 

57500 
57500 
57500 
57500 


MAX~WAIT 


0 


0 
0 
0 
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ENTRIES TIME*LEN 


25 0 
25 0 
25 0 
25 0 


TOT-WAIT 
0 


0 
0 
0 
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